Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Several parquet reader/writer regressions #17941

Merged
merged 8 commits into from
Jul 31, 2024
Merged

fix: Several parquet reader/writer regressions #17941

merged 8 commits into from
Jul 31, 2024

Conversation

coastalwhite
Copy link
Collaborator

No description provided.

@coastalwhite coastalwhite changed the title Fix: Several parquet reader/writer regressions fix: Several parquet reader/writer regressions Jul 30, 2024
@github-actions github-actions bot added fix Bug fix python Related to Python Polars rust Related to Rust Polars and removed title needs formatting labels Jul 30, 2024
@codecov Codecov
Copy link

codecov bot commented Jul 30, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 71.15385% with 30 lines in your changes missing coverage. Please review.

Project coverage is 80.39%. Comparing base (82b6388) to head (c45e177).
Report is 7 commits behind head on main.

Files Patch % Lines
...et/src/arrow/read/deserialize/fixed_size_binary.rs 10.52% 17 Missing ⚠️
...olars-parquet/src/arrow/read/deserialize/nested.rs 87.80% 5 Missing ⚠️
crates/polars-parquet/src/parquet/error.rs 0.00% 3 Missing ⚠️
...s-parquet/src/arrow/read/deserialize/binary/mod.rs 0.00% 2 Missing ⚠️
...s/polars-parquet/src/arrow/read/deserialize/mod.rs 60.00% 2 Missing ⚠️
crates/polars-parquet/src/arrow/write/pages.rs 88.88% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main   #17941      +/-   ##
==========================================
+ Coverage   80.35%   80.39%   +0.03%     
==========================================
  Files        1492     1494       +2     
  Lines      196330   196480     +150     
  Branches     2813     2817       +4     
==========================================
+ Hits       157764   157952     +188     
+ Misses      38045    38008      -37     
+ Partials      521      520       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

This was linked to issues Jul 31, 2024
1.3.0 regression when reading all-null DECIMAL(19,0) column @ parquet file exported by AWS Redshift #17929
Closed
Add tests for outer nullability of structs in Parquet #17595
Closed
Panic when creating simple nested dataframe with numpy #17531
Open
@coastalwhite coastalwhite removed a link to an issue Jul 31, 2024
Panic when creating simple nested dataframe with numpy #17531
Open
2 tasks
@coastalwhite coastalwhite linked an issue Jul 31, 2024 that may be closed by this pull request
Polars 1.3.0 throws parquet errors when reading (I think) categorical columns in settings that 1.1.0 does not. #17931
Closed
2 tasks
@coastalwhite coastalwhite marked this pull request as ready for review July 31, 2024 11:51
This was linked to issues Jul 31, 2024
File out of specification: The max_value of statistics MUST be plain encoded when writing nested parquet with Rust engine #17948
Closed
Polars 1.3.0 fails to collect scanned Parquet containing struct columns #17933
Closed
@ritchie46 ritchie46 merged commit 880e2e6 into pola-rs:main Jul 31, 2024
25 checks passed
@fzyzcjy
Copy link

fzyzcjy commented Aug 1, 2024

Hi, I wonder when will it be released? Thanks!

@ritchie46
Copy link
Member

Now!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ritchie46 ritchie46 ritchie46 approved these changes

@c-peters c-peters Awaiting requested review from c-peters c-peters is a code owner

@alexander-beedie alexander-beedie Awaiting requested review from alexander-beedie alexander-beedie is a code owner

@MarcoGorelli MarcoGorelli Awaiting requested review from MarcoGorelli MarcoGorelli is a code owner

@reswqa reswqa Awaiting requested review from reswqa reswqa is a code owner

@orlp orlp Awaiting requested review from orlp orlp is a code owner

Assignees
No one assigned
Labels
fix Bug fix python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Milestone
No milestone
3 participants
@coastalwhite @fzyzcjy @ritchie46